{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CPSC 330 Lecture 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outline:\n",
    "\n",
    "- 👋\n",
    "- **Turn on recording**\n",
    "- Announcements (5 min)\n",
    "- Activity: explaining `GridSearchCV` (15 min)\n",
    "- Principles of good explanations (15 min)\n",
    "- Break (5 min)\n",
    "- ML and decision-making (5 min)\n",
    "- Decision-making activity (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- When communicating about applied ML, tailor an explanation to the intended audience.\n",
    "- Apply best practices of technical communication, such as bottom-up explanations and reader-centric writing.\n",
    "- Given an ML problem, analyze the decision being made and the objectives.\n",
    "- Avoid the pitfall of thinking about ML as coding in isolation; build the habit of relating your work to the surrounding context and stakeholders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements (5 min)\n",
    "\n",
    "- hw5 grades posted this morning, hw6 grades coming very soon.\n",
    "- hw8 posted, due Monday 11:59pm.\n",
    "  - If your dataset is too big, feel free to subset; see https://piazza.com/class/kb2e6nwu3uj23?cid=564\n",
    "  - If you're having trouble finding datasets, feel free to start a Piazza thread; it's fine if you use the same dataset as another classmate/group\n",
    "- I will drop your lowest two assignment grades: https://piazza.com/class/kb2e6nwu3uj23?cid=560\n",
    "- The next 3 lectures will involve activities based on Google Docs\n",
    "  - I understand from the initial survey in September that there are a (very small) number of you who are unable to access Google Docs\n",
    "  - Sorry about that! You will still be able to see the results because I'll be sharing my screen, you just won't be able to post to the document.\n",
    "- Almost there, hang in there!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attribution\n",
    "\n",
    "The content of this lecture is adapted from [DSCI 542](https://github.com/UBC-MDS/DSCI_542_comm-arg), created by [David Laing](https://davidklaing.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why should I care about effective communication?\n",
    "\n",
    "- Most ML practitioners work in an organization with >1 people.\n",
    "- There will very likely be stakeholders other than yourself.\n",
    "- Those people need to understand what you're doing because:\n",
    "  - their state of mind may change the way you do things (see below)\n",
    "  - your state of mind may change the way they do things (interpreting your results)\n",
    "- In my experience, ML suffers from some particular communication issues:\n",
    "  - overstating one's results / unable to articulate the limitations\n",
    "  - unable to explain the predictions\n",
    "  - and the reason is: these things are actually very hard to explain!\n",
    "    - Why did CatBoost make that prediction?\n",
    "    - Can we trust test error?\n",
    "    - What does it mean if `predict_proba` outputs 0.9?\n",
    "    - Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activity: explaining `GridSearchCV` (15 min)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are two possible explanations of `GridSearchCV` pitched to different audiences. Read them both and then follow the instructions at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine learning algorithms, like an airplane's cockpit, typically involve a bunch of knobs and switches that need to be set.\n",
    "\n",
    "![](https://i.pinimg.com/236x/ea/43/f3/ea43f3c7f3a8c92d884ce012c77628fd--cockpit-gauges.jpg)\n",
    "\n",
    "For example, check out the documentation of the popular random forest algorithm [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html). Here's a list of the function arguments, along with their default values (from the documentation):\n",
    "\n",
    "> class sklearn.ensemble.RandomForestClassifier(n_estimators=100, criterion='gini', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=None, random_state=None, verbose=0, warm_start=False, class_weight=None, ccp_alpha=0.0, max_samples=None)\n",
    "\n",
    "Holy cow, that's a lot of knobs and switches! As a machine learning practitioner, how am I supposed to choose `n_estimators`? Should I leave it at the default of 100? Or try 1000? What about `criterion` or `class_weight` for that matter? Should I trust the defaults?\n",
    "\n",
    "Enter [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) to save the day. The general strategy here is the choose the settings that perform best on the specific task of interest. So I can't say `n_estimators=100` is better than `n_estimators=1000` without knowing what problem I'm working on. For a specific problem, you usually have a numerical score that measures performance. `GridSearchCV` is part of the popular [scikit-learn](https://scikit-learn.org/) Python machine learning library. It works by searching over various settings and tells you which one worked best on your problem. \n",
    "\n",
    "The \"grid\" in \"grid search\" comes from the fact that tries all possible combinations on a grid. For example, if you want it to consider setting `n_estimators` to 100, 150 or 200, and you want it to consider setting `criterion` to `'gini'` or `'entropy'`, then it will search over all 6 possible combinations in a grid of 3 possible values by 2 possible values: \n",
    "\n",
    "|                    | `criterion='gini'` | `criterion='entropy'` |\n",
    "|----------------------|--------|---------|\n",
    "| `n_estimators=100` |    1     |    2     |\n",
    "| `n_estimators=150` |    3     |    4     |\n",
    "| `n_estimators=200` |    5     |    6     |\n",
    "\n",
    "Here is a code sample that uses `GridSearchCV` to select from the 6 options we just mentioned. The problem being solved is classifying images of handwritten digits into the 10 digit categories (0-9). I chose this because the dataset is conveniently built in to scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'criterion': 'gini', 'n_estimators': 100}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# imports\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import datasets\n",
    "\n",
    "# load a dataset\n",
    "data = datasets.load_digits()\n",
    "X = data['data']\n",
    "y = data['target']\n",
    "\n",
    "# set up the grid search\n",
    "grid_search = GridSearchCV(RandomForestClassifier(random_state=123),\n",
    "                           param_grid={\n",
    "                                'n_estimators': [100, 150, 200],\n",
    "                                'criterion': ['gini', 'entropy']\n",
    "                           })\n",
    "\n",
    "# run the grid search\n",
    "grid_search.fit(X, y)\n",
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the output above, the grid search selected `criterion='gini', n_estimators=100`, which was one of our 6 options above (specifically Option 1).\n",
    "\n",
    "By the way, these \"knobs\" we've been setting are called [_hyperparameters_](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning) and the process of setting these hyperparameters automatically is called [_hyperparameter optimization_](https://en.wikipedia.org/wiki/Hyperparameter_optimization) or _hyperparameter tuning_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "~400 words, not including code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://medium.com/datadriveninvestor/an-introduction-to-grid-search-ff57adcc0998\n",
    "\n",
    "~400 words, not including code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion questions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What do you like about each explanation?\n",
    "- What do you dislike about each explanation?\n",
    "- What do you think is the intended audience for each explanation?\n",
    "- Which explanation do you think is more effective overall for someone on Day 1 of CPSC 330?\n",
    "- Each explanation has an image. Which one is more effective? What are the pros/cons?\n",
    "- Each explanation has some sample code. Which one is more effective? What are the pros/cons?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you're done reading, take ~5 min to consider the discussion questions above. Paste your answer to **at least one** of the above questions in the Google doc [link to be posted in the Zoom chat] under the appropriate question heading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Principles of good explanations (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Concepts *then* labels, not the other way around\n",
    "\n",
    "The first explanation start with an analogy for the concept (and the label is left until the very end):\n",
    "\n",
    "> Machine learning algorithms, like an airplane's cockpit, typically involve a bunch of knobs and switches that need to be set.\n",
    "\n",
    "In the second explanation, the first sentence is wasted on anyone who doesn't already know what \"hyperparameter tuning\" means:\n",
    "\n",
    "> Grid search is the process of performing hyper parameter tuning in order to determine the optimal values for a given model. \n",
    "\n",
    "The effectiveness of these different statements depend on your audience. \n",
    "\n",
    "See [this video](https://twitter.com/ProfFeynman/status/899963856549625858?s=20): \"I learned very early the difference between knowing the name of something and knowing something.\" -Richard Feynman."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bottom-up explanations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Curse of Knowledge](https://en.wikipedia.org/wiki/Curse_of_knowledge) leads to *top-down* explanations:\n",
    "\n",
    "![](img/top_down.png)\n",
    "\n",
    "- When you know something well, you think about things in the context of all your knowledge. \n",
    "- Those lacking the context, or frame of mind, cannot easily understand. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is another way: *bottom-up* explanations:\n",
    "\n",
    "![](img/bottom_up.png)\n",
    "\n",
    "When you're brand new to a concept, you benefit from analogies, concrete examples and familiar patterns.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### New ideas in small chunks\n",
    "\n",
    "The first explanation has a hidden conceptual skeleton:\n",
    "\n",
    "1. The concept of setting a bunch of values.\n",
    "2. Random forest example.\n",
    "3. The problem / pain point.\n",
    "4. The solution.\n",
    "5. How it works - high level.\n",
    "6. How it works - written example.\n",
    "7. How it works - code example.\n",
    "8. The name of what we were discussing all this time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Approach from all angles\n",
    "\n",
    "When we're trying to draw mental boundaries around a concept, it's helpful to see examples on all sides of those boundaries. If we were writing a longer explanation, it might have been better to show more, e.g.\n",
    "\n",
    "- Performance with and without hyperparameter tuning. \n",
    "- Other types of hyperparameter tuning (e.g. `RandomizedSearchCV`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reuse your running examples\n",
    "\n",
    "Effective explanations often use the same example throughout the text and code. This helps readers follow the line of reasoning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When experimenting, show the results asap\n",
    "\n",
    "The first explanation shows the output of the code, whereas the second does not. This is easy to do and makes a big difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interesting to you != useful to the reader (aka it's not about you)\n",
    "\n",
    "Here is something I deleted from my explanation:\n",
    "\n",
    "> Some hyperparameters, like `n_estimators` are numeric. Numeric hyperparameters are like the knobs in the cockpit: you can tune them continuously. `n_estimators` is numeric. Categorical hyperparameters are like the switches in the cockpit: they can take on (two or more) distinct values. `criterion` is categorical. \n",
    "\n",
    "It's a very elegant analogy! But is it helpful?\n",
    "\n",
    "And furthermore, what is my hidden motivation for wanting to include it? Elegance, art, and the pursuit of higher beauty? Or _making myself look smart_? So maybe another name for this principle could be **It's not about you.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "REMINDER TO RESUME RECORDING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ML and decision-making (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- There is often a wide gap between what people care about and what ML can do.\n",
    "- To understand what ML can do, let's think about what **decisions** will be made using ML. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decisions involve a few key pieces\n",
    "\n",
    "- The **decision variable**: the variable that is manipulated through the decision.\n",
    "  - E.g. how much should I sell my house for? (numeric)\n",
    "  - E.g. should I sell my house? (categorical)\n",
    "- The decision-maker's **objectives**: the variables that the decision-maker ultimately cares about, and wishes to manipulate indirectly through the decision variable.\n",
    "  - E.g. my total profit, time to sale, etc.\n",
    "- The **context**: the variables that mediate the relationship between the decision variable and the objectives.\n",
    "  - E.g. the housing market, cost of marketing it, my timeline, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How does this inform you as an ML practitioner?\n",
    "\n",
    "Questions you have to answer:\n",
    "\n",
    "- Who is the decision maker?\n",
    "- What are their objectives?\n",
    "- What are their alternatives?\n",
    "- What is their context?\n",
    "- What data do I need?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision-making activity (15 min)\n",
    "\n",
    "Consider the avocado price dataset from hw7. Let's say you work for Whole Foods, and they are wondering whether they should order more avocados this week or wait until next week.\n",
    "\n",
    "Answer the following questions:\n",
    "\n",
    "1. What are your decision variable(s) here?\n",
    "2. Is the decision numeric or categorical? What are the alternatives? \n",
    "3. What are the objective(s)?\n",
    "\n",
    "and then\n",
    "\n",
    "4. What data do you need here?\n",
    "5. What output might you show them from the model you trained in hw7?\n",
    "6. How does the output connect to the decisions?\n",
    "7. How would you present your results? What would you advise?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take 5-10 min for this activity, and then we'll discuss afterwards. Paste your answer to **at least one** of the above questions in the Google doc [link to be posted in the Zoom chat] under the appropriate question heading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "- Principles of effective communication\n",
    "  - Concepts then labels, not the other way around.\n",
    "  - Bottom-up explanations.\n",
    "  - New ideas in small chunks.\n",
    "  - Examples from all angles.\n",
    "  - Reuse your running examples.\n",
    "  - When experimenting, show the results asap.\n",
    "  - It's not about you.\n",
    "- Decision-making\n",
    "  - Decision variables, objectives, and context.\n",
    "  - How does ML fit in?\n",
    "  \n",
    "Next class we'll talk about communicating probabilities in your predictions, and we'll also talk about principles of effective visualizations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if we have extra time, we can do some parts of Lecture 24."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
